test(inkless:consolidation): adjust tests to address failures - #728
test(inkless:consolidation): adjust tests to address failures#728viktorsomogyi wants to merge 2 commits into
Conversation
3f7ea8d to
270a1cc
Compare
d2c5d85 to
50eead0
Compare
50eead0 to
9532a7f
Compare
jeqo
left a comment
jeqo
left a comment
There was a problem hiding this comment.
LGTM, just a suggestion on leveraging max.message.bytes instead of segment.bytes -- either one should work though.
| "remote.storage.enable": "true", | ||
| "min.insync.replicas": 2, | ||
| "segment.bytes": 1048576, | ||
| "segment.bytes": 2097152, |
There was a problem hiding this comment.
Non-blocker: Wonder if we can get the same effect by lowering max.message.bytes instead, so segments keep rolling at the current cadence and the retention/prune paths stay as exercised as before. 512 KiB on a 1 MiB segment leaves a 512 KiB fetch budget (two fetches per segment), which should be plenty for these datasets.
Batching is safe: VerifiableProducer only overrides request.timeout.ms, so batch.size stays at the 16 KiB default -- well inside the headroom.
| "segment.bytes": 2097152, | |
| "segment.bytes": 1024 * 1024, | |
| "max.message.bytes": 512 * 1024, |
There was a problem hiding this comment.
I think we can make this change, almost finished rerunning the tests.
Consolidation fetch requests segment.bytes minus max.message.bytes so one extra batch still fits the follower segment. Default max.message.bytes is 1048588 (1 MiB plus record overhead), so the old 1 MiB setting left a 1-byte fetch and starved the fetcher. 2 MiB leaves about 1 MiB of fetch budget. segment.ms still rolls on time.
9532a7f to
3ebe24e
Compare
2 participants
Consolidation fetch requests segment.bytes minus max.message.bytes so one extra batch still fits the follower segment. Default max.message.bytes is 1048588 (1 MiB plus record overhead), so the old 1 MiB setting left a 1-byte fetch and starved the fetcher.
2 MiB leaves about 1 MiB of fetch budget. segment.ms still rolls on time.